ftD  697798 


A  FHRL-TR  49-102 


FIELD  TEST  OF  THE  WEIGHTED  AIRMAN  PROMOTION  SYSTEM: 
PHASE  II.  VALIDATION  OF  THE  SYSTEM  FOR 
GRADES  E-4  THROUGH  E-7 


By 

Janos  B.  Koptyay 


PERSONNEL  RESEARCH  DIVISION 
Lac&iamt  A3r  Foraa  Bast.  Texas 


May  1909 


DEC  8  1968 


This  document  hat  been  ajprovod  for  public  release  and  sale: 
its  distribution  is  unlimced- 


rn 

■  i  B  ■ 

Ill'Bi 

i  ■  l  i 

I  !lil|  | 

Reproducecl  by  the 

CLEAR  1MGHOU5E 
for  federal  Scientific  &  Technical 
Information  Springfield  Va.  22151 


AIR  FORCE  SYSTEMS  COMMAND 

/ 

BROOKS  AIR  FORCE  BASE,  TEXASr 


NOTICE 

When  US  Government  drawings,  specifications,  or  other  data  are  used 
for  any  purpose  other  than  a  definitely  related  Government 
procurement  operation,  the  Government  thereby  incurs  no 
responsibility  nor  any  obligation  whatsoever,  and  the  fact  that  the 
Government  may  have  formulated,  furnished,  or  in  any  way  supplied 
the  said  drawings,  specifications,  or  other  data  is  not  to  be  regarded  by 
implication  or  otherwise,  as  in  any  manner  licensing  the  holder  or  any 
other  person  or  corporation,  or  conveying  any  rights  or  permission  to 
manuiacture,  use,  or  sell  any  patented  invention  that  may  in  any  way 
be  related  thereto. 


Composition:  Hope  Dc  La  Cn>7 
Mary  L.  Alvarado 
SMA:  3570 


AFHRL-TR4J9-102 


May  1968 


FIELD  TEST  OF  THE  WEIGHTED  AIRMAN  PROMOTION  SYSTEM: 
PHASE  II.  VAUDATION  OF  THE  SYSTEM  FOR 
GRADES  £-4  THROUGH  E-* 


By 

Janos  B.  Koylyiy 


TMs  document  bat  been  approved  for  public  ideate  and  tale; 
its  dirtribnrion  it  unlimited. 


PERSONNEL  RESEARCH  Dl  VISION 
AIR  FORCE  HUMAN  RESOURCES  LABORATORY 
AIR  FORCE  SYSTEMS  COMMAND 
LaddRid  Ak  Fosca  Bms,  Tinas 


FOREWORD 


This  research  was  completed  under  Project  6323,  Personnel  Management  Research 
and  Development;  Task  632305,  Development  of  Statistical  and  Mathematical  Procedures 
to  Facilitate  Personnel  Research. 

This  report  has  been  reviewed  and  is  approved. 


FJL  McLanathan,  Lt  Col,  USAF 
Chief,  Personnel  Research  Division 


ABSTRACT 


A  weighted  factors  promotion  system  was  field-tested  and  validated  using  data  from 
the  FY  69-B  promotion  cycle  of  the  Alaskan  Air  Command.  The  final  sample  included 
2,290  promotion-eligible  airmen  in  grades  E-3  through  E-6.  The  weighted  factors 
composite  score  excluding  a  promotion  board  component  gave  airmen  the  same  relative 
rank  within  selected  Air  Force  Specialties  as  did  the  promotion  board  evaluations  under 
the  present  operational  system.  Overlaps  between  the  two  ranks  imply  promotion  of  the 
same  individuals  by  both  systems.  However,  inconsistencies  and  unexplainable 
discrepancies  in  the  ranking  by  promotion  board  scores  were  found  in  some  few 
instances.  It  was  concluded  that,  within  the  specialties  analyzed,  practically  all  the 
individuals  promoted  by  the  board  system  would  also  have  been  promoted  under  the 
weighted  factors  system.  If  it  can  be  assumed  that  the  sample  was  representative  of  the 
Air  Force-wide  population  of  promotion-eligible  airmen  in  grades  E-3  through  E-6,  then  it 
can  be  further  assumed  that  the  weighted  factors  system  provides  a  valid  airman 
promotion  system  in  which  the  selection  criteria  are  visible  and  equitable. 


SUMMARY 


Kopiy  ay,  J.B.  Field  test  of  the  weighted  airman  promotion  system:  Phase  II.  Validation  of  the  system  for 
grades  E-4  through  E-7.  AFHRL-TR-69-102.  Lackland  AFB,  Tex.:  Personnel  Research  Division,  Air 
Force  Human  Resources  Laboratory,  May  1969. 

Problem 

In  response  to  a  research  requirement  levied  by  the  Deputy  Chief  of  Staff  for  Personnel, 
Headquarters,  United  States  Air  Force,  the  Personnel  Research  Division  developed  a  model  for  a  new 
airman  promotion  system.  The  purpose  of  this  new  promotion  system  was  to  provide  more  visibility  to 
each  airman  in  terms  of  numeric  values  on  selected  variables  indicating  their  relative  performance  on  these 
variables.  This,  in  tum,  provides  visible  evidence  and  information  about  the  reasons  for  non-promotion  in 
terms  of  easily  interpretable  scores. 

The  new  promotion  system  (Weighted  Airman  Promotion  System)  was  field-tested  in  the  Alaskan  Air 
Command  (Koplyay,  1969).  One  of  the  major  purposes  of  this  field  test  was  to  obtain  validation  data.  The 
operational  definition  of  validation  was  assumed  to  be  the  relationship  between  the  Weighted  Airman 
Promotion  System  and  the  existing  Promotion  Board  system  in  terms  of  the  percentage  of 
promotion-eligible  individuals  who  would  have  been  promoted  by  both  systems. 

Approach 

To  ascertain  the  predictive  efficiency  of  the  Weighted  Airman  Promotion  System,  a  validity  index  was 
computed  by  identifying  those  individuals  who  would  have  been  selected  for  promotion  by  both  the 
Composite  Score  of  the  new  system  and  the  Board  Score  of  the  existing  system.  This  validity  index  was 
computed  for  the  entire  sample  of  2,290  airmen  by  grade  (555  in  grade  E-3, 850  in  grade  E-4,  514  in  grade 
E-5,  and  371  in  grade  E-6),  and  was  further  analyzed  by  grade  within  selected  specialties. 

Results 

The  validity  indices  ranged  from  43.5  percent  (grade  E-6)  to  91.4  percent  (grade  E-3)  for  the  entire 
sample.  These  results,  however,  are  lower-limit  estimates  for  the  validity  of  the  new  system,  since  the 
existing  promotion  system  operates  on  the  basis  of  differential  quotas  assigned  to  the  various  specialties. 
The  validity  indices  within  grade  and  selected  specialties  ranged  from  84  percent  to  100  percent  with 
two-thirds  of  the  specialties  analyzed  having  validity  indices  of  100  percent. 

Conclusions 

The  Weighted  Airman  Promotion  System  gave  most  airmen  the  same  relative  ranking  within  their 
specialty  as  did  the  Promotion  Board  system.  In  other  words,  with  a  limited  number  of  exceptions,  the 
same  individuals  would  have  been  promoted  using  either  system. 

If  the  sample  in  the  study  can  be  considered  to  be  representative  of  the  Air  Force-wide  population  of 
airmen  in  grades  E-3  through  E-6,  it  can  be  further  assumed  that  the  Weighted  Airman  Promotion  System  is 
not  only  a  visible  system  but  it  is  valid  in  the  sense  of  the  operational  definition  of  validity  used  in  this 
study. 

This  summary  was  prepared  by  J.B.  Koplyay,  Statistical  Analysis  Branch,  Personnel  Research 
Division,  Air  Force  Human  Resources  Laboratory. 
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FIELD  TEST  OF  THE  WEIGHTED  AIRMAN  PROMOTION  SYSTEM: 
PHASE  II.  VALIDATION  OF  THE  SYSTEM  FOR 
GRADES  E4  THROUGH  E-7 


L  INTRODUCTION 

The  Weighted  Airman  Promotion  System 
was  developed,  field-tested  in  the  Alaskan  Air 
Command,  and  evaluated  in  response  to  a  research 
requirement  levied  by  the  Deputy  Chief  of  Staff 
for  Personnel,  Headquarters,  United  States  Air 
Force.  Briefly,  the  system  provides  for  selection  of 
airmen  for  promotion  to  grades  E4  through  E-7 
on  the  basis  of  seven  weighted  factors  designed  to 
insure  visibility  and  equitable  application  of  the 
selection  criteria.  One  of  the  seven  factors  is  a 
promotion  board  evaluation.  The  extent  to  which 
inclusion  of  this  board  save  as  a  component  in  the 
proposed  system  changed  the  r dative  standing  of 
promotion-eligible  individuals  was  discussed  in  the 
report  of  the  first  phase  of  this  study  (Kopiyay, 
1969).  In  general,  the  initial  findings  indicated 
that  indusion  of  the  board  component  made  only 
an  insignificant  difference  in  the  ranking  of  airmen 
who  were  eligible  for  promotion.  The  seven  factors 
and  their  computations  or  weights  are  shown  in 
the  appendix. 

One  of  the  major  purposes  of  the  Alaskan 
Air  Command  field  test  of  the  proposed  weighted 
factors  promotion  system  was  to  obtain  validation 
data.  Validation  is  a  very  general  concept,  and  it 
can  be  interpreted  in  a  variety  of  ways. 

In  this  report,  the  operational  definition  of 
validation  is  assumed  to  be  the  relationship 
between  the  weighted  factors  composite  save, 
exduding  the  average  board  score,  and  the  actual 
promotion  actions.  Since  promotions  under  tire 
existing  system  are  based  on  promotion  board 
scores,  validation  of  the  weighted  factors  system 
dealt  with  analyses  of  relationships  between  the 
weighted  factors  system  and  the  promotion  board 
system.  That  is,  if  the  same  individuals  were 
identified  for  promotion  by  the  weighted  factors 
system  as  were  actually  promoted  under  the 
existing  system,  then  the  weighted  factors 
composite  score  can  be  considered  a  valid 
indicator  (predictor)  of  promotion.  In  this 
context,  it  is  always  true  that  any  unreliability  in 
the  criterion  variable  places  an  upper  limit  on  the 
validity  of  any  predictor.  If  the  promotion  board 
score  has  a  less  than  perfect  relationship  with 
promotion  actions,  it  is  impossible  to  develop  any 


kind  of  system  to  duplicate  it.  In  this  report,  two 
general  indicators  are  used  to  express  the  validity 
of  the  proposed  weighted  factors  system.  One  is 
the  comparison  of  the  weighted  factors  composite 
scores  with  actual  promotion  actions.  The  other 
reflects  directly  the  relationship  between  the 
proposed  system  and  the  promotion  board  system. 

This  report  focuses  primarily  on  three 
specific  areas: 

1.  The  relationship  between  promotion 
board  score  and  promotion  action. 

2.  The  agreement  between  rank  standing  on 
the  weighted  factors  composite  score  excluding 
the  board  score  and  rank  standing  on  the 
promotion  board  score  for  the  total  sample  and 
for  selected  Control  Air  Force  Specialties. 

3.  The  analysis  of  groups  with  large 
discrepancies  between  rank  on  the  weighted 
fact  on  composite  score  and  rank  on  the  board 
scrte. 


IL  PROCEDURE 

Description  of  the  Sample 

The  Alaskan  Air  Command  provided  data 
for  2,835  airmen  in  grades  E-3  through  E-6  who 
were  eligible  for  promotion  in  the  FY  69-B  cyde. 
All  factors  except  the  Promotion  Fitness 
Examination  Scores  were  provided  on  punched 
cards  by  Consolidated  Base  Personnel  Offices  at 
Elmendorf  and  Eielson  Air  Force  Bases.  The 
Promotion  Fitness  Examination  was  administered 
separately,  and  scores  were  added  to  the  record  of 
each  airman  in  the  sample.  Promotion  board  scores 
were  given  in  raw  score  form  as  the  sum  of  tire 
ratings  of  individual  promotion  board  members. 
There  were  both  three-man  and  five-man  boards. 

Treatment  of  the  Data 

A  number  of  steps  were  taken  to  obtain 
usable  and  meaningful  data  for  analysis. 

1.  Promotion  Fitness  Examination  scores 
were  added  to  the  data  tape  containing  the  card 
images  of  punched  cards  provided  by  the  Alaskan 
Air  Command. 
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2.  Records  were  excluded  from  further 
analyses  if  Specialty  Knowledge  Test  scores  or 
Promotion  Fitness  Examination  scores  were 
missing,  or  if  non-matching  and  uncorrectable 
serial  numbers  and  names  were  detected  during  the 
merging  of  input  data  tapes.  As  a  result,  2,290 
airmen  (S5S  in  grade  E-3, 850  in  grade  E-4,  514  in 
grade  E-5,  and  371  in  grade  E-6)  were  retained  in 
the  study  for  further  analyses. 

3.  Promotion  board  raw  scores  were 
converted  to  a  scale,  with  a  100-point  maximum 
score  (see  Appendix).  Each  raw  score  was  divided 
by  the  appropriate  number  of  board  members,  and 
the  quotient  was  multiplied  by  10.  The  resulting 
score  gave  an  average  board  score  for  each  airman. 
(Before  adopting  the  procedure  of  generating  the 
average  board  score,  a  preliminary  analysis  of  the 
data  had  shown  that  the  variability  of  the  resulting 
scores  would  be  approximately  constant  among 
grades.)  In  case  of  ties  in  average,  board  scores,  an 
attempt  was  mvie  to  break  these  ties  by 
consideration  of  the  lime-in-Grade,  Airman 
Performance  Report,  and  Decorations  scores,  in 
that  order.  (This  was  in  accordance  with  the  policy 
of  the  promotion  board,  but  it  did  not  eliminate 
ties  altogether.)  The  average  board  score,  then,  was 
added  to  each  airman’s  record.  This  score  is 
referred  to  simply  as  the  Board  Score  in  the 
remainder  of  this  report. 

4.  Time-in-Grade  and  Time-in-Service  scores 
were  recomputed  using  Date  of  Rank,  Total  Active 
Federal  Military  Service  Date,  and  the  date  1 
January  1969  for  airmen  in  grade  E-3  and  1  May 
1969  for  airmen  in  grades  E-4  through  E-6.  The 
recomputed  values  were  distributed  against  those 
supplied  by  the  Alaskan  Air  Command.  With  the 
exception  of  one  case,  which  was  correctable,  the 
recomputed  values  agreed  with  the  original  values. 

5.  Total  weighted  factors  composite  scores 
were  computed  by  adding  the  relevant  factors  (see 
Appendix)  in  two  ways:  («)  by  excluding  the 
average  board  score  and  (b)  by  including  the 
average  board  score.  Results  of  the  Phase  I  analysis 
had  indicated  conclusively  that  inclusion  of  a 
board  component  in  the  weighted  factors 
composite  would  introduce  only  trivial  changes  in 
a  comparable  composite  which  excluded  a  board 
component.  Thus,  the  weighted  factors  composite 
scores  including  the  board  component  were 
computed  and  recorded,  but  were  not  used  in  any 
of  the  analyses.  The  weighted  factors  composite 
score  excluding  the  board  component  is  referred 
to  as  the  Composite  Score  in  the  remainder  of  this 
report. 


6.  Within  each  grade,  airmen  were  ranked 
on  both  the  Composite  Score  and  the  Board  Score. 
These  created  ranks  were  added  to  each  record  on 
the  data  tape.  In  general,  the  higher  the  score  on  a 
particular  variable,  the  lower  the  corresponding 
rank  value,  For  example,  a  rank  value  of  1  was 
assigned  to  the  individual  who  had  the  highest 
score.  Thus,  high  Composite  Scores  and  high 
Board  Scores  are  reflected  in  low  rank  values. 

7.  During  the  analysis,  a  list  was  obtained  of 
the  promotion-eligible  airmen  who  were  actually 
promoted.  This  information  was  added  to  the  data 
tape  as  an  additional  score  of  1  if  the  airman  was 
promoted  and  0  if  he  was  not.  The  variable  thus 
created  served  as  the  indicator  of  promotion 
action. 

8.  Data  available  for  the  weighted  factors 
promotion  system  were  distributed  by  single 
factors  and  by  combinations  of  several  factors. 
From  the  distribution  by  Control  Air  Force 
Specialty  Code,  groups  with  sufficient  frequencies 
were  selected  as  subgroups  for  more  detailed 
analysis. 

9.  Since  one  of  the  major  factors  in  actual 
promotion  is  the  quota  allocated  to  a  particular 
specialty,  grouping  of  cases  by  specialties  within 
grade  seemed  to  be  essential  for  meaningful 
analysis.  Thus,  airmen  were  re-ranked  within  their 
particular  specialties  and  grade  on  both  the 
Composite  Score  and  the  Board  Score.  These  new 
ranks  were  used  only  when  the  groups  were 
analyzed  within  specialties. 

in.  ANALYSES 

Relationship  Between  Board  Score 
and  Promotion  Action 

Traditionally,  promotions  have  been  made 
primarily  on  the  basis  of  promotion  board  scores; 
however,  if  the  promotion  quota  is  either  very 
high  or  very  low,  the  board  score  is  less  of  a 
determining  factor.  When  the  quota  is  high,  for 
example,  eligible  airmen  within  a  wide  range  of 
board  scores  from  high  to  low  are  selected.  If  the 
quota  is  low,  eligible  airmen  with  both  high  and 
low  board  scores  are  not  selected,  while  only  those 
with  the  very  highest  scores  are  selected.  When 
many  eligible  airmen  receive  tied  board  scores,  but 
the  quota  does  not  permit  all  such  airmen  to  be 
promoted,  a  promotion/non-promotion  decision 
must  be  made.  Promotion  boards  have  generally 
accomplished  this  by  taking  a  “second  look”  at 
the  selection  folders  of  the  airmen  with  tied  board 
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scores  to  consider  other  criteria  in  making  the  se- 
lections.  For  these  reasons,  it  was  anticipated  that 
the  relationship  might  be  less  than  perfect  between 
average  board  scores  and  promotion/non- 
promotion  actions.  To  obtain  empirical  evidence 
on  this  question,  the  relationship  between  the 
Board  Score  and  the  actual  promotion  action  was 
determined  within  each  of  several  specialties  and 
for  the  four  grade  levels  involved. 

The  most  widely  used  technique  to  ascertain 
relationships  is  computation  of  a  coefficient  of 
correlation.  Correlation  coefficients,  obtained  as  a 
by-product  of  the  regression  analyses  performed 
and  reported  separately  under  Phase  I  of  this  study 
(Koplyay,  1969),  are  shown  in  Table  1. 
Interpreting  correlation  coefficients,  particularly 
when  the  correlation  is  between  a  continuous 
variable  (such  as  Board  Score)  and  a  binary 
variable  (such  as  promotion/non-promotion),  is 
extremely  difficult  because  the  correlation  not 
only  depends  on  the  strength  of  the  relationship 
between  the  two  variables,  but  it  is  also  affected 
by  other  factors  such  as  the  “split”  on  the  binary 
variable  (e.g.,  the  proportion  of  promoted  airmen 
to  non-promoted  airmen)  and  the  number  of 
unique  values  on  the  non-dichotomized  variable 
(e.g.,  the  number  of  ties  in  Board  Score).  For 
example,  in  the  case  of  grade  E-3,  specialty  702,  it 
was  known  that  24  of  the  25  eligible  airmen  were 
actually  promoted.  With  the  exception  of  one 
airman  in  this  grade  and  specialty,  all  eligible 
airmen  were  promoted  regardless  of  their  Board 
Score.  Thus,  the  computed  correlation  indicates  a 
weak  relationship  between  Board  Score  and 
promotion  action.  The  correlation  coefficients 
indicated  in  Table  1,  therefore,  should  be 
interpreted  with  caution  and  in  light  of  any 
circumstances  which  might  yield  spurious  results. 


From  Table  1 ,  it  is  apparent  that  the  Board 
Scores  correlated  less  than  perfectly  with  the 
actual  promotions  within  grade.  Correlations 
within  grade  by  selected  specialties  ranged  from 
.1989  to  .8029,  showing  both  relatively  low  and 
relatively  high  correlations.  As  already  suggested, 
however,  these  correlations  reflect  a  somewhat 
blurred  picture  of  the  precise  relationship  between 
the  Board  Score  and  actual  promotion.  The  low 
values  could  be  a  result  of  any  one  or  all  of  several 
reasons:  (a)  extreme  split  between  promoted  and 
non-promoted  subgroups,  (f>)  tied  Board  Scores 
among  those  competing  for  promotions,  and  (c) 
inversions  in  which  eligible  airmen  with  lower 
Board  Sco/es  were  promoted  over  airmen  with 
higher  Board  Scores.  Further  analysis  of  the  data 
indicated  that  there  were  only  two  inversions; 
hence,  this  reason  can  be  eliminated  as  a  factor  to 
account  for  the  observed  low  correlations.  There 
were,  however,  occasional  extreme  splits  and  a 
high  incidence  of  ties  in  Board  Score  (as  discussed 
in  Phase  I  of  this  study),  and  apparently  these  are 
the  major  factors  accounting  for  the  low 
correlations  which  do  exist. 

There  is  one  other  possible  explanation  for 
the  less  than  perfect  relationship  between  Board 
Scores  and  the  actual  promotion  actions.  The 
results  of  Phase  I  of  the  study  had  revealed 
statistically  significant  differences  between  the 
means  of  the  Board  Scores  for  the  two  bases, 
Elmendorf  and  Eielson,  for  airmen  in  grade  E-3. 
Since  these  differences  only  existed  for  grade  E-3, 
and  the  promotion/non-promotion  split  for  this 
grade  was  consistently  extreme  across  specialties, 
it  was  felt  that  possible  effects  of  differences  in 
Board  Score  means  (which  generally  were 
numerically  small  although  statistically  significant) 
could  not  have  been  completely  separated  from 


Table  1.  Correlations  Between  Board  Score  and  Promotion  Action  Within  Each  Pay  Grade 
for  Total  Sample  and  for  Selected  Specialties 
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the  advene  effect  of  the  extreme  splits  on  the 
correlation  between  Board  Score  and  actual 
promotion  action.  Furthermore,  subsequent 
analyses  indicated  that,  with  the  exception  of  two 
inversions  (i.e.,  cases  in  which  airmen  with  lower 
Board  Scores  were  promoted  over  airmen  with 
higher  Board  Scores),  all  airmen  who  were 
promoted,  regardless  of  their  base  of  assignment, 
had  Board  Scores  as  high  as  or  higher  than  those 
who  were  not  promoted.  This  finding  seemed  to 
justify  the  conclusion  that  extreme  promo- 
tion/non-promotion  splits  are  mainly  responsible 
for  the  less  than  perfect  correlation  between  Board 
Score  and  promotion  action  as  shown  in  this 
study. 

Agreement  Between  Promotion 
Ptafictkms  and  Actual  Promotion 
Outcomes  Within  Pay  Grades 

Another  phase  of  the  analysis  dealt  with  the 
validation  of  the  Composite  Score  against  actual 
selection,  and  then  comparison  of  that  relationship 
with  the  relationship  between  Board  Score  and 
promotion  action.  To  appraise  the  validity  of  the 
proposed  weighted  factors  composite  score,  it  was 
assumed  that  the  system  had,  in  fact,  been  used  as 
the  basis  for  determining  promotion/non¬ 
promotion  decisions.  The  resulting  outcomes 
derived  from  this  hypothetical  application  of  the 
system  would  then  be  considered  “predictions”  of 
the  selection/non-selection  outcomes  as  indicated 
by  the  actual  promotion  actions.  Thus,  if  an 
eligible  airman  would  have  been  promoted  by  the 
proposed  system  and  was,  in  fact,  promoted,  then 
the  proposed  system  provided  a  correct  prediction 
of  the  promotion  outcome.  The  validity  of  the 
proposed  system  is  operationally  defined  for  the 
purpose  of  this  analysis  in  terms  of  the  number  of 
correct  promotion  predictions  and  the  proportion 
of  correct  promotion  predictions.  Ranks  on  the 


Composite  Score  were  used  as  the  basis  for 
selecting  individuals  to  receive  hypothetical 
promotions  under  the  proposed  system. 

To  ascertain  the  predictive  efficiency  of  the 
proposed  system,  a  validity  index  was  computed 
by  identifying  those  individuals  who  would  have 
been  selected  for  promotion  by  both  the 
Composite  Score  and  the  Board  Score.  The  ratio 
of  the  number  of  such  individuals  to  the  actual 
number  of  promotions  is  operationally  defined  as 
promotion  overlap.  Table  2  summarizes  the 
correct  promotion  predictions,  or  promotion 
overlap,  for  the  Composite  Score  by  pay  grade. 

An  example  drawn  from  the  data  in  Table  2 
will  illustrate  the  predictive  validation.  For  grade 
E-5,  the  number  of  correct  promotion  predictions 
of  the  proposed  system  was  determined  by 
distributing  and  ranking  the  Composite  Scores  of 
the  514  E-5  airmen.  With  a  promotion  quota  of 
89,  the  89  airmen  with  the  highest  scores  would 
presumably  have  been  selected  for  promotion  and 
the  remaining  425  non-selected.  Of  the  514  airmen 
in  grade  E-5,  56  airmen  were  selected  for 
promotion  by  both  the  Composite  Score  and  the 
Board  Score.  This  represents  a  promotion  overlap 
of  62.9  percent  (or  56/89).  In  other  words,  62.9 
percent  of  those  airmen  whose  rank  on  the  Board 
Score  was  better  than  or  equal  to  the  promotion 
quota  of  89  were  also  ranked  among  the  top  89  on 
the  Composite  Score.  Although  this  does  not  mean 
that  these  56  airmen  received  the  same  rank  on 
both  Composite  Score  and  Promotion  Score,  the 
end  result  of  promotion  selection  would  be  the 
same.  That  is,  one  airman  could  have  been  ranked 
1  on  the  Composite  Score  and  89  on  the  Board 
Score;  nevertheless,  he  would  have  been  selected 
for  promotion  by  either  system  since  the  quota 
was  89. 


Table  2.  Correct  Promotion  Predictions  by  Composite  Score 
Within  Each  Pay  Grade 
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If  there  were  one  single  promotion  quota  for 
each  pay  grade,  rather  than  different  quotas  for 
different  specialties  as  is  the  case,  the  data  in  Table 
2  would  adequately  answer  the  question  of  valid¬ 
ity  of  the  proposed  system.  In  general,  the  results 
in  Table  2  indicate  only  a  moderate  level  of  valid¬ 
ity  of  the  system  for  actual  promotion/non¬ 
promotion  outcomes,  with  the  percentage  of 
correct  promotion  predictions  ranging  from  43. 5 
percent  to  91.4  percent.  Furthermore,  this  overlap 
in  promotion  predictions  is  less  than  70  percent 
for  three  of  the  four  grades  in  question.  These 
results,  however,  should  be  considered  as  lower- 
limit  estimates  for  the  validity  of  the  proposed 
system  and  should  be  interpreted  with  caution 
since  the  existing  promotion  system  does  operate 
on  the  basis  of  differential  quotas.  All  the  airmen 
within  a  pay  grade  are  ranked  on  promotion  board 
score,  and  then  the  quota  for  each  specialty  is 
filled  going  down  the  list  from  high  score  to  low. 
Under  this  system,  it  is  quite  possible  that  airmen 
with  high  promotion  board  scores  will  not  be 
promoted  and  airmen  with  lower  board  scores  will 
be  promoted. 

To  clarify  this,  let  us  assume  that  a 
particular  airman  ranks  10  on  promotion  board 
score,  and  five  of  his  fellow  airmen  in  the  same 
specialty  rank  9  or  better  on  the  board  score. 
Suppose,  further,  that  this  particular  specialty  has 
a  quota  of  four  promotions.  The  airman  in 
question  will  not  be  promoted  since  there  are  five 
persons  in  his  specialty  who  rank  better  than  he 
does  on  the  board  score.  Now,  let  us  assume  that 
another  airman,  who  is  in  the  same  pay  grade  as 
the  first  airman  but  in  a  different  specialty,  ranks 
55  on  the  board  score.  Assuming,  further,  that  this 
second  airman’s  specialty  has  a  quota  of  five 
promotions  and  that  no  other  airman  in  his  group 
has  a  rank  better  than  55  on  the  board  score,  this 
airman  will  be  promoted  since  his  board  score  rank 
is  the  highest  within  his  specialty.  The  promotion 
outcome  is  favorable  for  the  second  airman  and 
unfavorable  for  the  first  airman  in  spite  of  the  fact 
that  he  had  a  better  promotion  board  score  in 
terms  of  the  comparison  across  pay  grade. 

Agreement  Between  Promotion  Predictions 
and  Actual  Promotion  Outcomes 
Within  Selected  Specialties 

Because  of  the  differences  by  specialty,  a 
further  step  was  undertaken  to  analyze  the 
relationship  between  the  Composite  Score  and 
Board  Score  in  terms  of  correct  promotion 
predictions  within  particular  specialties.  Two  new 
variables  were  created:  rank  on  the  Composite 


Score  and  rank  on  the  Board  Score  within 
specialties. 

The  distribution  of  airmen  by  Control  Air 
Force  Specialty  Codes  did  not  provide  an  adequate 
sample  for  all  the  specialties  in  the  study.  Most  of 
the  specialties  did  not  have  enough  individuals 
within  each  pay  grade  to  permit  a  meaningful 
analysis.  It  was  possible,  however,  to  select  six 
specialties  with  a  large  enough  sample  within  each 
pay  grade  for  further  analysis.  The  same 
procedures  as  used  in  the  comparison  for  the  total 
sample  were  used  for  the  within-specialty  analyses. 
The  only  exception  was  that  each  airman  was 
re-ranked  on  the  Composite  Score  and  the  Board 
Score  within  his  specialty. 

Table  3  summarizes  the  promotion 
predictions  within  grade  and  by  selected 
specialties.  These  results  clearly  imply  that  there 
was  a  very  high  degree  of  agreement  in  promotion 
prediction  between  the  Composite  Score  and  the 
Board  Score.  The  lowest  promotion  overlap  was 
83 33  percent,  and  in  16  out  of  the  24  specialty- 
by-grade  combinations,  the  agreement  was  100 
percent.  One  must  conclude  that  the  Composite 
Score  would  have  promoted  the  same  individuals 
in  the  16  specialties  analyzed  as  did  the  Board 
Score.  Within  the  remaining  8  specialties,  the 
overlap  of  predicted  promotions  ranged  from  8333 
percent  to  96.84  percent. 

Analysis  of  the  specialties  in  which 
agreement  fell  short  of  100  percent  indicates  that 
the  policy  used  in  breaking  tied  promotion  board 
scores  was  not  applied  with  complete  consistency. 
This  may  account  for  a  small  number  of  cases 
which  were  predicted  for  promotion  by  the  Board 
Score  but  not  by  the  Composite  Score,  and  vice 
versa.  Also,  it  appears  that  in  some  few  cases  the 
board  rated  an  individual  high  if  his  Airman 
Performance  Report  variable  was  high,  regardless 
of  low  scores  on  other  variables.  In  short,  the  same 
factors  which  contribute  to  the  lack  of  complete 
visibility  of  the  selection  criteria  under  the 
promotion  board  system  probably  also  account  for 
the  less  than  perfect  relationship  between  the 
Composite  Score  and  the  Board  Score. 

Sample  Cases  of  Inconsistencies  Between 
Composite  Scores  and  Board  Scores 

To  illustrate  the  occasional  inconsistencies 
of  the  promotion  board  scores  in  light  of  the 
components  of  the  weighted  factors  system,  the 
scores  and  ranks  of  four  airmen  in  grade  E-4  are 
compared  in  Table  4.  It  is  apparent  from  the  table 
that  Airman  A’s  performance  was  dearly  much 
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Table  3.  Correct  Promotion  Predictions  by  Composite  Score 
for  Selected  Specialties  Within  Each  Pay  Grade 
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Table  4.  Comparison  of  Composite  Scon  and  Board  Score  Variables 
for  Four  Sample  Caaes 


Variable 

Sample  Casas 

Airman  A 

Airman  a 

Airman  C 

Airman  O 

Specialty  Knowledge  Test  Score 

80 

30 

95 

60 

Promotion  Fitness  Examination  Score 

85 

15 

95 

75 

Decorations  Score 

0 

0 

0 

0 

Airman  Performance  Report  Score 

125 

128 

135 

135 

Time-in-Grade  Score 

44 

10 

12 

7 

Time-in-Service  Score 

23 

7 

8 

6 

Composite  Score  Rank 

2 

792 

8 

Board  Score  Rank 

759 

77 

230 

2 

Promotion  Action 

0 

1 

1 

1 

superior  to  Airman  B*s.  In  fact.  Airman  B  scored 
very  poorly  in  general.  Yet,  Airman  A  ranked  759 
on  the  Board  Score  and  was  not  promoted. 
Airman  B,  on  the  other  hand,  ranked  77  on  the 
Board  Score  (792  on  the  Composite  Score)  and 
was  promoted. 

Airman  C  obtained  scores  far  superior  to 
those  of  Airman  D.  Although  both  were 
promoted,  C  ranked  230  on  the  Board  Score  (with 
superior  performance  on  the  Composite  Score);  D 
ranked  2  on  the  Board  Score  (with  poor 
performance  on  the  Composite  Score). 

The  inconsistencies  which  are  present  in  the 
sample  indicate  that  the  board  must  be  evaluating 
some  ‘Invisible”  additional  factors  besides  those 
included  in  the  weighted  factors  composite  in 
these  cases.  Since  rankings  on  the  Board  Score  in 
the  total  sample  within  grade  (across  specialties) 
are  not  influenced  by  quotas  assigned  to  the 
specialties,  inferences  about  inconsistencies  in 
Board  Score  rankings  are  valid.  Although  it  is  clear 
in  these  cases  that  the  Board  Score  reflects  a 
weighting  of  information  from  the  selection  folder 
which  is  not  included  in  the  Composite  Score,  it 
should  be  reemphasized  that  the  actual  promotion 
outcome  is  a  function  not  only  of  the  Board  Score 
but  also  of  the  quota  assigned  to  a  specialty.  Thus, 
while  a  low  Board  Score  decreases  the  likelihood 
of  promotion,  and  a  high  Board  Score  increases 
the  likelihood  of  promotion,  it  does  not 
necessarily  follow  that  a  high  rank-value  on  the 
Board  Score  results  in  non-promotion  and  a  low 
rank-value  on  the  Board  Score  guarantees 


promotion.  That  is,  a  rank  (within  grade)  of  230 
(Airman  C  in  Table  4)  could  have  represented  a 
relative  rank  (within  specialty)  good  enough  to  be 
promoted  based  on  the  quota  for  that  specialty. 

Analyis  of  Rank  Discrepancies 

Hie  sample  within  each  grade  was  divided 
into  four  mutually  exclusive  groups  based  on  ranks 
on  the  Composite  Score  and  the  Board  Score: 

Group  1.  Individuals  whose  ranks  on  both 
scores  were  less  than  or  equal  to  the 
promotion  quota. 

Group  2.  Individuals  whose  rank  on  the 
Composite  Score  was  less  than  or  equal  to 
the  promotion  quota,  and  whose  rank  on  the 
Board  Score  was  greater  than  the  promotion 
quota. 

Group  3.  Individuals  whose  rank  on  the 
Composite  Score  was  greater  than  the 
promotion  quota,  and  whose  rank  on  the 
Board  Score  was  less  than  or  equal  to  the 
promotion  quota. 

Group  4.  Individuals  whose  ranks  on  both 
scores  were  greater  than  the  promotion 
quota. 

Groups  1  and  4  are  the  “agreement”  groups; 
groups  2  and  3  are  the  “discrepancy”  groups.  The 
latter  groups,  2  and  3,  were  analyzed  on  the 
components  of  the  weighted  factors  composite 
score.  The  results  of  these  analyses  are  summarized 
in  Table  5. 


Table  5.  Analysis  of  Weighted  Factors  for  Discrepancy  Groups  in  Which  Composite  Score 
and  Board  Score  Ranks  Were  Not  in  Agreement  on  Promotion  Action 


Comparison  of  the  groups  in  terms  of  the  six 
components  of  the  weighted  factors  composite 
score  reveals  the  following  findings. 

1.  Specialty  Knowledge  Test  Scores.  Group 
2  (high  Composite  Scores  but  low  Board  Scores) 
was  superior  to  group  3  (low  Composite  Scores 
but  high  Board  Scores)  in  all  grades.  The  means 
differed  by  more  than  20  points  except  in  grade 
E-5  where  the  difference  was  13.9  points.  In  all 
cases,  the  differences  were  statistically  significant. 

2.  Promotion  Fitness  Examination  Scores. 
The  means  of  group  2  were  superior  to  the  means 
of  group  3  in  all  grades.  The  differences  ranged 
from  23.6  points  (E-6)  to  44.5  points  (E-3). 

3.  Decorations  Scores.  There  were  no 
statistically  significant  differences  between  the 
means  of  group  2  and  group  3. 

4.  Airman  Performance  Report  Scores. 
There  was  a  statistically  significant  difference 
between  means  of  group  2  and  group  3  in  favor  of 
group  3  in  all  grades.  These  differences,  however, 
were  relatively  small  numerically  (18.7  for  E-3, 5.0 
for  E-4, 6.2  for  E-5,  and  2.9  for  E-6). 

5.  Time-in-Crade  Scores.  With  the  exception 
of  grade  E-3,  means  of  group  2  and  group  3 
differed  significantly  in  favor  of  group  2  in  all 
grades. 

6.  Time- in-Service  Scores.  Means  of  group  2 
and  group  3  differed  significantly  in  favor  of  group 
2  in  all  grades  except  grade  E-3.  The  difference 
was  relatively  small  but  statistically  significant. 

With  the  exception  of  the  Decorations 
factor,  where  the  two  discrepancy  groups  were 
comparable,  and  the  Airman  Performance  Report 
factor,  where  a  small  but  significant  difference  in 
favor  of  group  3  was  found,  group  2  was  superior 
on  all  factors. 


IV.  SUMMARY 

The  weighted  fact  on  promotion  system  was 
fidd-tr  ted  with  the  cooperation  of  the  Alaskan 
Air  Command.  From  among  2,835 
promotion-eligible  airmen  at  Hmendorf  and 
Eietson  Air  Force  Bases,  a  final  sample  of  2,290 
airmen  in  grades  E-3  through  E-6  was  obtained. 
Data  were  analyzed  to  determine  the  relationship 
between  the  promotion  board  score  (Board  Score) 
and  the  weighted  factors  composite  score 
excluding  the  board  :rore  (Composite  Score).  In 
addition,  actual  promotion  actions  woe  related  to 
promotion  board  scores.  Overlaps  between  rank  on 
the  Board  Score  and  rank  on  the  Composite  Score 


were  analyzed  for  the  entire  sample  within  each 
grade  and  for  selected  specialties.  Discrepancies  in 
ranks  for  a  few  individual  cases  were  further 
analyzed. 

The  results  of  the  various  analyses  are 
summarized,  and  several  conclusions  are  drawn 
regarding  the  relationship  between  the  existing 
airman  promotion  system  based  on  board 
evaluations  and  the  weighted  factors  promotion 
system. 

1.  The  relationship  between  the  Board 
Score  and  actual  promotion  was  less  than  perfect. 
Correlations  ranged  between  .471  and  .562  for  the 
total  sample.  For  selected  specialties,  the  range 
was  from  .199  to  .803.  /.'though  ties  on  the  Board 
Scores  may  have  been  n  .  tially  responsible  for  the 
observed  low  to  moderate  correlations,  it  is  felt 
that  the  different  promotion  quotas  for  different 
specialties  probably  account  for  additional 
variability  in  the  relationship  between  Board  Score 
and  actual  promotion. 

2.  Promotion  overlap  between  Composite 
Score  and  Board  Score  ranged  from  43  percent  for 
grade  E-6  to  91  percent  for  grade  E-3  for  the  total 
sample.  Further  analysis  within  specialties  resulted 
in  almost  complete  overlap.  When  airmen  were 
re-ranked  on  the  Composite  Score  and  on  the 
Board  Score  within  their  specialties,  the  overlap 
was  100  percent  in  16  out  of  the  24  groups 
analyzed.  The  overlap  ranged  between  84  percent 
and  %  percent  in  the  other  eight  groups.  This  does 
not  necessarily  imply  that  the  ranks  on  Board 
Score  and  Composite  Score  were  the  same.  It  does 
imply,  however,  that  when  re-ranked  within  a 
particular  specialty,  practically  the  same  airmen 
would  have  been  selected  for  promotion  by  both 
the  Board  Score  alone  and  the  Composite  Score 
excluding  the  Board  Score.  Thus,  it  is  apparent 
that  the  proposed  weighted  factors  promotion 
system  is  highly  valid  in  the  sense  that  it  provides 
results  which  are  in  dose  agreement  with  the 
operational  system. 

3.  Analyses  of  individual  cases  indicated 
occasional  inconsistencies  on  the  part  of  the 
promotion  board  in  assigning  Board  Scores  on  the 
basis  of  the  factors  included  in  the  weighted 
factors  composite  score.  In  some  cases,  airmen 
with  superior  scores  on  these  factors  were  ranked 
lower  by  the  board  than  others  whose  scores 
reflected  poor  performance. 

4.  Most  airmen  either  ranked  high  on  both 
the  Board  Score  and  the  Composite  Score,  or  they 
ranked  low  on  both.  A  relatively  small  number  of 
cases  ranked  high  on  the  Composite  Score  and  low 
on  the  Board  Score,  while  another  small  number 
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ranked  high  on  the  Board  Score  and  low  on  the 
Composite  Score.  Comparison  of  these 
rank-discrepancy  groups  revealed  that,  with  the 
exception  of  the  Airman  Performance  Report 
variable  and  the  Decorations  variable,  the  average 
score  on  each  factor  was  higher  for  the  group  in 
which  the  rank  discrepancy  was  in  favor  of  the 
Composite  Score  than  in  the  reverse  discrepancy 
group.  The  Airman  Performance  Report  mean  was 
slightly  higher  in  the  group  which  ranked  high  on 
the  Board  Score  and  low  on  the  Composite  Score. 
However,  the  margin  by  which  the  Airman  Per¬ 
formance  Report  factor  was  higher  was  relatively 
slight  compared  to  the  large  differences  on  other 
factors  revealed  when  the  rank-discrepancy  groups 
were  analyzed.  The  Decorations  factor  was 
approximately  equal  in  the  two  groups. 

To  summarize  the  overall  findings,  then,  the 
weighted  factors  composite  score  excluding  the 
average  board  score  gave  most  airmen  the  same 
relative  ranking  within  their  specialty  as  did  the 
promotion  board  score.  In  other  words,  the  same 
individuals  would  have  been  promoted  using  either 
system.  On  the  other  hand,  when  all  promotion- 
eligible  airmen  within  a  pay  grade  were  compared 
with  each  other,  there  were  indications  of 
inconsistencies  in  the  promotion  board’s  judgment 
of  the  relative  merits  of  performance  in  arriving  at 
the  promotion/non-promotion  decision.  There  was 
a  definite  lack  of  visibility  of  selection  factors  in  a 
few  cases  where  airmen  with  superior  Composite 


Scores  were  given  low  Board  Scores.  This  does  not 
imply  that  the  promotion  board  had  no  valid 
reason  for  its  judgment.  It  does  suggest,  however, 
that  the  reason  would  not  be  readily  apparent  to 
an  airman  under  the  current  operational  board 
process. 

Validation  of  the  weighted  factors 
promotion  system  was  accomplished  by  ranking 
the  individuals  on  their  Composite  Scores  and 
comparing  these  ranks  to  actual  promotion 
outcomes.  It  was  concluded  from  the  results  that 
the  weighted  factors  promotion  system  would 
have  promoted  almost  the  same  individuals  within 
the  specialties  analyzed  as  were  actually  selected 
by  the  promotion  boards.  If  the  sample  in  the 
study  can  be  considered  to  be  repiesentative  of  the 
Air  Force-wide  population  of  promotion-eligible 
airmen  in  grades  E-3  through  E-6,  it  can  be  further 
assumed  that  the  weighted  factors  system  provides 
a  valid  promotion  system  in  which  the  selection 
criteria  are  visible. 
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Human  Resources  Laboratory,  April  1969. 
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APPENDIX:  AIRMAN  PROMOTION  SELECTION  FACTORS  AND  POINTS* 


Grades E-4  through  B-7 


Sal  act  Ion  Factor 

Computation  \Mth 

Board  Seen 

ComputaUon  Wthout 

Board  Score 

Maximum 

Point! 

Ftfonrtt|t 

Maximum 

Point* 

Faroe  ntaaa 

Specialty  Knowledge  Test  (SKT)  Score 

95 

17 

95 

21 

Promotion  Fitness  Examination  (PFE)  Score 

95 

17 

95 

21 

Time-In-Service  (TIS)  Score 

40 

7 

40 

9 

Time-In-Grade  (TIG)  Score 

60 

11 

60 

13 

Decoration  Score 

25 

5 

25 

6 

Airman  Performance  Reports  (APR) 

135 

25 

135 

30 

Board  Evaluation 

100 

18 

550 

100 

450 

100 

Explanation  of  Factors 

The  SKT  and  PFE  will  be  administered  annually. 

Points  for  the  SKT  and  PFE  scores  will  be  actual  percentile  scores  obtained  in  these  tests  (in  5-point 
increments). 

Time-in-Service  win  be  computed  by  multiplying  yean  of  Total  Active  Federal  Military  Service  by  2.  Less 
than  6  months  will  count  as  1  point;  over  6  months  will  count  a  full  year,  2  points.  A  cutoff  score  of  40 
points,  for  20  yean  TAFMS,  has  been  established. 

Time-in-Grade  will  be  computed  at  the  rate  of  Vi  point  per  month  up  to  a  maximum  of  120  months,  60 
points;  1 5  days  or  less  will  be  dropped,  16  or  more  will  count  as  a  full  month. 

Decorations  will  be  assigned  points  according  to  their  order  of  precedence.  The  maximum  number  of  points 
attainable  is  25.  Decorations  will  count  for  promotion  regardless  of  the  military  service  in  which  they  were 
earned. 

The  Airman  Performance  Report  score  is  obtained  by  multiplying  the  overall  evaluation  mean  by  15.  The 
mean  is  based  on  reports  for  a  5-year  period  prior  to  the  eligibility  date,  not  to  exceed  ten  reports. 

The  Board  Score  will  be  based  on  a  review  by  the  board  that  concentrates  on  those  items  not  previously 
weighted;  e.g.,  education  level  and  efforts  to  improve  self  in  terms  of  formal  education,  technical 
knowledge,. etc.  Reduced  selection  folder  will  consist  of  Category  A  favorable  communications,  APR  word 
picture,  and  pages  2  and  4  of  the  Air  Force  Form  7. 


aFrom  Kopiy  ay,  1969,  p.  11. 
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